Content item selection and click probability determination based upon accidental click events

ABSTRACT

In an example, sets of event information associated with events may be identified. The events may include intentional click events, accidental click events and/or skip events. Accidental click probabilities associated with the accidental click events and/or the skip events may be determined. Machine learning model training may be performed, using the sets of event information associated with the events and labels associated with the events, to generate a first machine learning model. The labels may include second labels associated with the intentional click events and/or third labels associated with the accidental click events and/or the skip events. The second labels may correspond to an intentional click classification. The third labels may be based upon the accidental click probabilities. Click probabilities associated with content items may be determined using the first machine learning model. A content item may be selected for presentation via a client device based upon the click probabilities.

RELATED APPLICATION

This application claims priority to and is a continuation of U.S.application Ser. No. 17/346,339, filed on Jun. 14, 2021, entitled“CONTENT ITEM SELECTION AND CLICK PROBABILITY DETERMINATION BASED UPONACCIDENTAL CLICK EVENTS”, which is incorporated by reference herein inits entirety.

BACKGROUND

Many services, such as websites, applications, etc. may provideplatforms for viewing media. For example, a user may interact with aservice. While interacting with the service, selected media may bepresented to the user automatically. Some of the media may beadvertisements advertising products and/or services associated with acompany.

SUMMARY

In accordance with the present disclosure, one or more computing devicesand/or methods are provided. In an example, a first plurality of sets ofevent information, associated with a first plurality of events, may beidentified. The first plurality of sets of event information maycomprise a second plurality of sets of event information associated witha plurality of intentional click events of the first plurality ofevents, a third plurality of sets of event information associated with aplurality of accidental click events of the first plurality of eventsand a fourth plurality of sets of event information associated with aplurality of skip events of the first plurality of events. An accidentalclick event of the plurality of accidental click events may correspondto an event in which a content item is presented via a client device anda selection (e.g., a click) of the content item is received via theclient device, where the selection is determined (e.g., predicted) to beaccidental. A plurality of accidental click probabilities associatedwith a second plurality of events may be determined. The secondplurality of events may comprise the plurality of accidental clickevents and the plurality of skip events. A first accidental clickprobability of the plurality of accidental click probabilities may beassociated with a first accidental click event of the plurality ofaccidental click events. The first accidental click probability may bedetermined based upon a first set of event information associated withthe first accidental click event. A second accidental click probabilityof the plurality of accidental click probabilities may be associatedwith a first skip event of the plurality of skip events. The secondaccidental click probability may be determined based upon a second setof event information associated with the first skip event. Machinelearning model training may be performed, using the first plurality ofsets of event information associated with the first plurality of eventsand a first plurality of labels associated with the first plurality ofevents, to generate a first machine learning model. The first pluralityof labels may comprise a second plurality of labels associated with theplurality of intentional click events and a third plurality of labelsassociated with the second plurality of events. Labels of the secondplurality of labels may correspond to an intentional clickclassification. Labels of the third plurality of labels may be basedupon the plurality of accidental click probabilities. The thirdplurality of labels may comprise a first label, associated with thefirst accidental click event, based upon the first accidental clickprobability. The third plurality of labels may comprise a second label,associated with the first skip event, based upon the second accidentalclick probability. A request for content associated with a client devicemay be received. A plurality of click probabilities associated with aplurality of content items may be determined using the first machinelearning model in response to receiving the request for content. A firstcontent item of the plurality of content items may be selected forpresentation via the client device based upon the plurality of clickprobabilities.

In an example, a first plurality of sets of event information,associated with a first plurality of events, may be identified. Thefirst plurality of sets of event information may comprise a secondplurality of sets of event information associated with a plurality ofintentional click events of the first plurality of events and a thirdplurality of sets of event information associated with a plurality ofskip events of the first plurality of events. A plurality of accidentalclick probabilities associated with the plurality of skip events may bedetermined. A first accidental click probability of the plurality ofaccidental click probabilities may be associated with a first skip eventof the plurality of skip events. The first accidental click probabilitymay be determined based upon a first set of event information associatedwith the first skip event. Machine learning model training may beperformed, using the first plurality of sets of event informationassociated with the first plurality of events and a first plurality oflabels associated with the first plurality of events, to generate afirst machine learning model. The first plurality of labels may comprisea second plurality of labels associated with the plurality ofintentional click events and a third plurality of labels associated withthe plurality of skip events. Labels of the second plurality of labelsmay correspond to an intentional click classification. Labels of thethird plurality of labels may be based upon the plurality of accidentalclick probabilities. The third plurality of labels may comprise a firstlabel, associated with the first skip event, based upon the firstaccidental click probability. A request for content associated with aclient device may be received. A plurality of click probabilitiesassociated with a plurality of content items may be determined using thefirst machine learning model in response to receiving the request forcontent. A first content item of the plurality of content items may beselected for presentation via the client device based upon the pluralityof click probabilities.

In an example, a first plurality of sets of event information,associated with a first plurality of events, may be identified. Thefirst plurality of sets of event information may comprise a secondplurality of sets of event information associated with a plurality ofintentional click events of the first plurality of events and a thirdplurality of sets of event information associated with a plurality ofaccidental click events of the first plurality of events. A plurality ofaccidental click probabilities associated with the plurality ofaccidental click events may be determined. A first accidental clickprobability of the plurality of accidental click probabilities may beassociated with a first accidental click event of the plurality ofaccidental click events. The first accidental click probability may bedetermined based upon a first set of event information associated withthe first accidental click event. Machine learning model training may beperformed, using the first plurality of sets of event informationassociated with the first plurality of events and a first plurality oflabels associated with the first plurality of events, to generate afirst machine learning model. The first plurality of labels may comprisea second plurality of labels associated with the plurality ofintentional click events and a third plurality of labels associated withthe plurality of accidental click events. Labels of the second pluralityof labels may correspond to an intentional click classification. Labelsof the third plurality of labels may be based upon the plurality ofaccidental click probabilities. The third plurality of labels maycomprise a first label, associated with the first accidental clickevent, based upon the first accidental click probability. A request forcontent associated with a client device may be received. A plurality ofclick probabilities associated with a plurality of content items may bedetermined using the first machine learning model in response toreceiving the request for content. A first content item of the pluralityof content items may be selected for presentation via the client devicebased upon the plurality of click probabilities.

DESCRIPTION OF THE DRAWINGS

While the techniques presented herein may be embodied in alternativeforms, the particular embodiments illustrated in the drawings are only afew examples that are supplemental of the description provided herein.These embodiments are not to be interpreted in a limiting manner, suchas limiting the claims appended hereto.

FIG. 1 is an illustration of a scenario involving various examples ofnetworks that may connect servers and clients.

FIG. 2 is an illustration of a scenario involving an exampleconfiguration of a server that may utilize and/or implement at least aportion of the techniques presented herein.

FIG. 3 is an illustration of a scenario involving an exampleconfiguration of a client that may utilize and/or implement at least aportion of the techniques presented herein.

FIG. 4 is a flow chart illustrating an example method for determiningclick probabilities associated with content items and/or selectingcontent for transmission to devices.

FIG. 5A is a component block diagram illustrating an example system fordetermining click probabilities associated with content items and/orselecting content for transmission to devices, where a client devicepresents and/or accesses a first webpage.

FIG. 5B is a component block diagram illustrating an example system fordetermining click probabilities associated with content items and/orselecting content for transmission to devices, where a client devicepresents a plurality of search results associated with a query.

FIG. 5C is a component block diagram illustrating an example system fordetermining click probabilities associated with content items and/orselecting content for transmission to devices, where a client devicetransmits a request to access a resource to a server.

FIG. 5D is a component block diagram illustrating an example system fordetermining click probabilities associated with content items and/orselecting content for transmission to devices, where a request forcontent is received.

FIG. 5E is a component block diagram illustrating an example system fordetermining click probabilities associated with content items and/orselecting content for transmission to devices, where a client devicepresents and/or accesses a fourth webpage displaying a content item.

FIG. 5F is a component block diagram illustrating an example system fordetermining click probabilities associated with content items and/orselecting content for transmission to devices, where a client devicepresents and/or accesses a seventh webpage in response to a selection ofa content item.

FIG. 5G is a component block diagram illustrating an example system fordetermining click probabilities associated with content items and/orselecting content for transmission to devices, where a client devicereturns to a fourth web page.

FIG. 5H is a component block diagram illustrating an example system fordetermining click probabilities associated with content items and/orselecting content for transmission to devices, where a machine learningtraining module performs machine learning model training to generate afirst machine learning model.

FIG. 5I is a component block diagram illustrating example training dataof an example system for determining click probabilities associated withcontent items and/or selecting content for transmission to devices.

FIG. 5J is a component block diagram illustrating an example system fordetermining click probabilities associated with content items and/orselecting content for transmission to devices, where a plurality ofaccidental click probabilities is determined using a first machinelearning model.

FIG. 5K is a component block diagram illustrating an example system fordetermining click probabilities associated with content items and/orselecting content for transmission to devices, where a machine learningtraining module performs machine learning model training to generate asecond machine learning model.

FIG. 5L is a component block diagram illustrating example training dataof an example system for determining click probabilities associated withcontent items and/or selecting content for transmission to devices.

FIG. 5M is a component block diagram illustrating an example system fordetermining click probabilities associated with content items and/orselecting content for transmission to devices, where a click probabilityis determined using a second machine learning model.

FIG. 6 is a component block diagram illustrating an example system fordetermining click probabilities associated with content items and/orselecting content for transmission to devices.

FIG. 7 is an illustration of a scenario featuring an examplenon-transitory machine readable medium in accordance with one or more ofthe provisions set forth herein.

DETAILED DESCRIPTION

Subject matter will now be described more fully hereinafter withreference to the accompanying drawings, which form a part hereof, andwhich show, by way of illustration, specific example embodiments. Thisdescription is not intended as an extensive or detailed discussion ofknown concepts. Details that are known generally to those of ordinaryskill in the relevant art may have been omitted, or may be handled insummary fashion.

The following subject matter may be embodied in a variety of differentforms, such as methods, devices, components, and/or systems.Accordingly, this subject matter is not intended to be construed aslimited to any example embodiments set forth herein. Rather, exampleembodiments are provided merely to be illustrative. Such embodimentsmay, for example, take the form of hardware, software, firmware or anycombination thereof.

1. Computing Scenario

The following provides a discussion of some types of computing scenariosin which the disclosed subject matter may be utilized and/orimplemented.

1.1. Networking

FIG. 1 is an interaction diagram of a scenario 100 illustrating aservice 102 provided by a set of servers 104 to a set of client devices110 via various types of networks. The servers 104 and/or client devices110 may be capable of transmitting, receiving, processing, and/orstoring many types of signals, such as in memory as physical memorystates.

The servers 104 of the service 102 may be internally connected via alocal area network 106 (LAN), such as a wired network where networkadapters on the respective servers 104 are interconnected via cables(e.g., coaxial and/or fiber optic cabling), and may be connected invarious topologies (e.g., buses, token rings, meshes, and/or trees). Theservers 104 may be interconnected directly, or through one or more othernetworking devices, such as routers, switches, and/or repeaters. Theservers 104 may utilize a variety of physical networking protocols(e.g., Ethernet and/or Fiber Channel) and/or logical networkingprotocols (e.g., variants of an Internet Protocol (IP), a TransmissionControl Protocol (TCP), and/or a User Datagram Protocol (UDP). The localarea network 106 may include, e.g., analog telephone lines, such as atwisted wire pair, a coaxial cable, full or fractional digital linesincluding T1, T2, T3, or T4 type lines, Integrated Services DigitalNetworks (ISDNs), Digital Subscriber Lines (DSLs), wireless linksincluding satellite links, or other communication links or channels,such as may be known to those skilled in the art. The local area network106 may be organized according to one or more network architectures,such as server/client, peer-to-peer, and/or mesh architectures, and/or avariety of roles, such as administrative servers, authenticationservers, security monitor servers, data stores for objects such as filesand databases, business logic servers, time synchronization servers,and/or front-end servers providing a user-facing interface for theservice 102.

Likewise, the local area network 106 may comprise one or moresub-networks, such as may employ differing architectures, may becompliant or compatible with differing protocols and/or may interoperatewithin the local area network 106. Additionally, a variety of local areanetworks 106 may be interconnected; e.g., a router may provide a linkbetween otherwise separate and independent local area networks 106.

In the scenario 100 of FIG. 1 , the local area network 106 of theservice 102 is connected to a wide area network 108 (WAN) that allowsthe service 102 to exchange data with other services 102 and/or clientdevices 110. The wide area network 108 may encompass variouscombinations of devices with varying levels of distribution andexposure, such as a public wide-area network (e.g., the Internet) and/ora private network (e.g., a virtual private network (VPN) of adistributed enterprise).

In the scenario 100 of FIG. 1 , the service 102 may be accessed via thewide area network 108 by a user 112 of one or more client devices 110,such as a portable media player (e.g., an electronic text reader, anaudio device, or a portable gaming, exercise, or navigation device); aportable communication device (e.g., a camera, a phone, a wearable or atext chatting device); a workstation; and/or a laptop form factorcomputer. The respective client devices 110 may communicate with theservice 102 via various connections to the wide area network 108. As afirst such example, one or more client devices 110 may comprise acellular communicator and may communicate with the service 102 byconnecting to the wide area network 108 via a wireless local areanetwork 106 provided by a cellular provider. As a second such example,one or more client devices 110 may communicate with the service 102 byconnecting to the wide area network 108 via a wireless local areanetwork 106 (and/or via a wired network) provided by a location such asthe user's home or workplace (e.g., a WiFi (Institute of Electrical andElectronics Engineers (IEEE) Standard 802.11) network or a Bluetooth(IEEE Standard 802.15.1) personal area network). In this manner, theservers 104 and the client devices 110 may communicate over varioustypes of networks. Other types of networks that may be accessed by theservers 104 and/or client devices 110 include mass storage, such asnetwork attached storage (NAS), a storage area network (SAN), or otherforms of computer or machine readable media.

1.2. Server Configuration

FIG. 2 presents a schematic architecture diagram 200 of a server 104that may utilize at least a portion of the techniques provided herein.Such a server 104 may vary widely in configuration or capabilities,alone or in conjunction with other servers, in order to provide aservice such as the service 102.

The server 104 may comprise one or more processors 210 that processinstructions. The one or more processors 210 may optionally include aplurality of cores; one or more coprocessors, such as a mathematicscoprocessor or an integrated graphical processing unit (GPU); and/or oneor more layers of local cache memory. The server 104 may comprise memory202 storing various forms of applications, such as an operating system204; one or more server applications 206, such as a hypertext transportprotocol (HTTP) server, a file transfer protocol (FTP) server, or asimple mail transport protocol (SMTP) server; and/or various forms ofdata, such as a database 208 or a file system. The server 104 maycomprise a variety of peripheral components, such as a wired and/orwireless network adapter 214 connectible to a local area network and/orwide area network; one or more storage components 216, such as a harddisk drive, a solid-state storage device (SSD), a flash memory device,and/or a magnetic and/or optical disk reader.

The server 104 may comprise a mainboard featuring one or morecommunication buses 212 that interconnect the processor 210, the memory202, and various peripherals, using a variety of bus technologies, suchas a variant of a serial or parallel AT Attachment (ATA) bus protocol; aUniform Serial Bus (USB) protocol; and/or Small Computer SystemInterface (SCI) bus protocol. In a multibus scenario, a communicationbus 212 may interconnect the server 104 with at least one other server.Other components that may optionally be included with the server 104(though not shown in the schematic diagram 200 of FIG. 2 ) include adisplay; a display adapter, such as a graphical processing unit (GPU);input peripherals, such as a keyboard and/or mouse; and a flash memorydevice that may store a basic input/output system (BIOS) routine thatfacilitates booting the server 104 to a state of readiness.

The server 104 may operate in various physical enclosures, such as adesktop or tower, and/or may be integrated with a display as an“all-in-one” device. The server 104 may be mounted horizontally and/orin a cabinet or rack, and/or may simply comprise an interconnected setof components. The server 104 may comprise a dedicated and/or sharedpower supply 218 that supplies and/or regulates power for the othercomponents. The server 104 may provide power to and/or receive powerfrom another server and/or other devices. The server 104 may comprise ashared and/or dedicated climate control unit 220 that regulates climateproperties, such as temperature, humidity, and/or airflow. Many suchservers 104 may be configured and/or adapted to utilize at least aportion of the techniques presented herein.

1.3. Client Device Configuration

FIG. 3 presents a schematic architecture diagram 300 of a client device110 whereupon at least a portion of the techniques presented herein maybe implemented. Such a client device 110 may vary widely inconfiguration or capabilities, in order to provide a variety offunctionality to a user such as the user 112. The client device 110 maybe provided in a variety of form factors, such as a desktop or towerworkstation; an “all-in-one” device integrated with a display 308; alaptop, tablet, convertible tablet, or palmtop device; a wearable devicemountable in a headset, eyeglass, earpiece, and/or wristwatch, and/orintegrated with an article of clothing; and/or a component of a piece offurniture, such as a tabletop, and/or of another device, such as avehicle or residence. The client device 110 may serve the user in avariety of roles, such as a workstation, kiosk, media player, gamingdevice, and/or appliance.

The client device 110 may comprise one or more processors 310 thatprocess instructions. The one or more processors 310 may optionallyinclude a plurality of cores; one or more coprocessors, such as amathematics coprocessor or an integrated graphical processing unit(GPU); and/or one or more layers of local cache memory. The clientdevice 110 may comprise memory 301 storing various forms ofapplications, such as an operating system 303; one or more userapplications 302, such as document applications, media applications,file and/or data access applications, communication applications such asweb browsers and/or email clients, utilities, and/or games; and/ordrivers for various peripherals. The client device 110 may comprise avariety of peripheral components, such as a wired and/or wirelessnetwork adapter 306 connectible to a local area network and/or wide areanetwork; one or more output components, such as a display 308 coupledwith a display adapter (optionally including a graphical processing unit(GPU)), a sound adapter coupled with a speaker, and/or a printer; inputdevices for receiving input from the user, such as a keyboard 311, amouse, a microphone, a camera, and/or a touch-sensitive component of thedisplay 308; and/or environmental sensors, such as a global positioningsystem (GPS) receiver 319 that detects the location, velocity, and/oracceleration of the client device 110, a compass, accelerometer, and/orgyroscope that detects a physical orientation of the client device 110.Other components that may optionally be included with the client device110 (though not shown in the schematic architecture diagram 300 of FIG.3 ) include one or more storage components, such as a hard disk drive, asolid-state storage device (SSD), a flash memory device, and/or amagnetic and/or optical disk reader; and/or a flash memory device thatmay store a basic input/output system (BIOS) routine that facilitatesbooting the client device 110 to a state of readiness; and a climatecontrol unit that regulates climate properties, such as temperature,humidity, and airflow.

The client device 110 may comprise a mainboard featuring one or morecommunication buses 312 that interconnect the processor 310, the memory301, and various peripherals, using a variety of bus technologies, suchas a variant of a serial or parallel AT Attachment (ATA) bus protocol;the Uniform Serial Bus (USB) protocol; and/or the Small Computer SystemInterface (SCI) bus protocol. The client device 110 may comprise adedicated and/or shared power supply 318 that supplies and/or regulatespower for other components, and/or a battery 304 that stores power foruse while the client device 110 is not connected to a power source viathe power supply 318. The client device 110 may provide power to and/orreceive power from other client devices.

In some scenarios, as a user 112 interacts with a software applicationon a client device 110 (e.g., an instant messenger and/or electronicmail application), descriptive content in the form of signals or storedphysical states within memory (e.g., an email address, instant messengeridentifier, phone number, postal address, message content, date, and/ortime) may be identified. Descriptive content may be stored, typicallyalong with contextual content. For example, the source of a phone number(e.g., a communication received from another user via an instantmessenger application) may be stored as contextual content associatedwith the phone number. Contextual content, therefore, may identifycircumstances surrounding receipt of a phone number (e.g., the date ortime that the phone number was received), and may be associated withdescriptive content. Contextual content, may, for example, be used tosubsequently search for associated descriptive content. For example, asearch for phone numbers received from specific individuals, receivedvia an instant messenger application or at a given date or time, may beinitiated. The client device 110 may include one or more servers thatmay locally serve the client device 110 and/or other client devices ofthe user 112 and/or other individuals. For example, a locally installedwebserver may provide web content in response to locally submitted webrequests. Many such client devices 110 may be configured and/or adaptedto utilize at least a portion of the techniques presented herein.

2. Presented Techniques

One or more computing devices and/or techniques for determining clickprobabilities associated with content items and/or selecting content fortransmission to devices are provided. For example, a first user (and/ora first client device associated with the first user) may access and/orinteract with a service, such as a browser, software, a website, anapplication, an operating system, etc. that provides a platform forviewing and/or downloading content from a server associated with acontent system. In some examples, in response to receiving a request forcontent associated with the first client device, the content system maydetermine click probabilities associated with a plurality of contentitems (e.g., advertisements, images, links, videos, etc.). The clickprobabilities may be used to select a content item, from the pluralityof content items, for presentation via the first client device. In someexamples, a click probability is representative of (e.g., comprises) aprobability of receiving a selection (e.g., a click) of a content itemin response to presenting the content item via the first client device(e.g., a probability that presentation of the content item via the firstclient device would be followed by a selection, such as a click, of thecontent item on the first client device).

In some examples, the click probabilities may be determined based upon afirst user profile associated with the first user and click events. Theclick events may correspond to selections of content items presented viaclient devices (e.g., the content items may be provided by the contentsystem in response to requests for content). For example, a first clickprobability associated with presenting a first content item via thefirst client device may be determined based upon the first user profile,click events associated with the first content item and/or user profilesassociated with users that performed the click events. The click eventsmay comprise accidental click events. An accidental click event maycorrespond to a selection of a content item that is determined (e.g.,predicted) to have been performed accidentally and/or unintentionally,such as when a user mistakenly clicks on an advertisement. An accidentalclick event may be detected and/or identified, such as using one or moreof the techniques herein, based upon a dwell time associated with aselection of a content item (e.g., the accidental click event may bedetected and/or identified based upon the dwell time being less than athreshold dwell time).

Some systems may not distinguish between the accidental click events andintentional click events when determining the first click probability.For example, the first click probability may be increased due to anassumed affinity and/or interest that the first user has towards thefirst content item that is assumed based upon the accidental clickevents associated with other users (e.g., other users with user profilessimilar to the first user profile of the first user). Since selections(e.g., clicks) of the accidental click events may be unintentional, theaccidental click events may not be reflective of an affinity and/orinterest of the other users towards the first content item, and thus,the first click probability determined using these systems may beinaccurate. Alternatively and/or additionally, treating the intentionalclick events and the accidental click events in the same way indetermining the first click probability may lead to an inaccuratedetermination of the first click probability. Accordingly,distinguishing between the accidental click events and the intentionalclick events (and/or using the accidental click events and theintentional click events in different ways) when determining the firstclick probability, such as using one or more of the techniques discussedherein, may lead to more accurate determination of the first clickprobability (e.g., a more accurate representation of a probability thatthe first content item is selected in response to presenting the firstcontent item via the first client device).

Other systems may filter the accidental click events from click eventsused to determine the first click probability. For example, thesesystems may remove the accidental click events from data used todetermine the first click probability, and/or may not account for theaccidental click events and/or accidental click probabilities whendetermining the first click probability. The first click probabilitydetermined using these systems may be inaccurate, such as due tounder-prediction of click probabilities. For example, although anaccidental click event may not indicate an affinity and/or interest of auser towards a content item, the accidental click event corresponds to aselection (e.g., a click) of a content item. Accordingly, determiningaccidental click probabilities and/or using the accidental clickprobabilities to determine the first click probability, such as usingone or more of the techniques discussed herein, may lead to moreaccurate determination of the first click probability.

Thus, in accordance with one or more of the techniques presented herein,a first plurality of sets of event information, associated with a firstplurality of events, may be identified. The first plurality of sets ofevent information may comprise a second plurality of sets of eventinformation associated with a plurality of intentional click events ofthe first plurality of events, a third plurality of sets of eventinformation associated with a plurality of accidental click events ofthe first plurality of events and/or a fourth plurality of sets of eventinformation associated with a plurality of skip events of the firstplurality of events. A plurality of accidental click probabilitiesassociated with a second plurality of events may be determined. In someexamples, the plurality of accidental click probabilities may bedetermined based upon events, such as accidental click events and/orskip events. For example, a first machine learning model may be trainedusing sets of event information associated with accidental click eventsand/or skip events, and/or the plurality of accidental clickprobabilities may be determined using the first machine learning model.The second plurality of events may comprise the plurality of accidentalclick events and/or the plurality of skip events. The plurality ofaccidental click probabilities may be determined based upon sets ofevent information, of the first plurality of sets of event information,associated with the second plurality of events. Machine learning modeltraining may be performed, using the first plurality of sets of eventinformation associated with the first plurality of events and a firstplurality of labels associated with the first plurality of events, togenerate a second machine learning model. The first plurality of labelsmay comprise a second plurality of labels associated with the pluralityof intentional click events and a third plurality of labels associatedwith the second plurality of events. Labels of the second plurality oflabels may correspond to an intentional click classification. Labels ofthe third plurality of labels may be based upon the plurality ofaccidental click probabilities. For example, a label (of the thirdplurality of labels) that is associated with an event of the secondplurality of events may be determined based upon an accidental clickprobability, of the plurality of accidental click probabilities,associated with the event. In an example, the label may be indicative ofa value that is equal to the accidental click probability. Alternativelyand/or additionally, the label may be indicative of a value that isdifferent than the accidental click probability (e.g., one or moreoperations, such as mathematical operations, may be performed using theaccidental click probability and one or more other values to determinethe value of the label). A request for content associated with a clientdevice may be received. A plurality of click probabilities associatedwith a plurality of content items may be determined using the secondmachine learning model in response to receiving the request for content.A first content item of the plurality of content items may be selectedfor presentation via the client device based upon the plurality of clickprobabilities.

It may be appreciated that by determining click probabilities based uponaccidental click probabilities, such as using one or more of thetechniques herein, the click probabilities may be determined moreaccurately. Alternatively and/or additionally, by training the firstmachine learning model using labels that are based upon accidental clickprobabilities, the first machine learning model may be used to determineclick probabilities with increased accuracy. For example, some systemsmay set labels associated with click events to a first value (e.g., 1)for use in training a machine learning model for determining clickprobabilities, regardless of whether the click events are accidentalclick events or intentional click events. By training the first machinelearning model using training data that differentiates betweenaccidental click events and intentional click events (e.g., by settinglabels associated with intentional click events to a value correspondingto an intentional click classification, such as 1, and setting labelsassociated with accidental click events to values based upon theaccidental click probabilities), in accordance with one or moreembodiments of the present disclosure, click probabilities determinedusing the first machine learning model may have an increased accuracy ascompared to the systems that set labels associated with click events tothe first value regardless of whether the click events are accidentalclick events or intentional click events. Alternatively and/oradditionally, by training the first machine learning model usingtraining data that comprises labels associated with skip events that arebased upon accidental click probabilities associated with the skipevents, in accordance with one or more embodiments of the presentdisclosure, click probabilities determined using the first machinelearning model may have an increased accuracy as compared to systemsthat set labels associated with skip events to a single value (e.g., 0)for training a machine learning model.

An embodiment of determining click probabilities associated with contentitems and/or selecting content for transmission to devices isillustrated by an example method 400 of FIG. 4 . A content system forpresenting content via devices may be provided. In some examples, thecontent system may be an advertisement system (e.g., an onlineadvertising system). Alternatively and/or additionally, the contentsystem may not be an advertisement system. In some examples, the contentsystem may provide content items (e.g., advertisements, images, links,videos, etc.) to be presented via pages associated with the contentsystem. For example, the pages may be associated with websites (e.g.,websites providing search engines, email services, news content,communication services, etc.) associated with the content system. Thecontent system may provide content items to be presented in (dedicated)locations throughout the pages (e.g., one or more areas of the pagesconfigured for presentation of content items). For example, a contentitem may be presented at the top of a web page associated with thecontent system (e.g., within a banner area), at the side of the web page(e.g., within a column), in a pop-up window, overlaying content of theweb page, etc. Alternatively and/or additionally, a content item may bepresented within an application associated with the content systemand/or within a game associated with the content system. Alternativelyand/or additionally, a user may be required to watch and/or interactwith the content item before the user can access content of a web page,utilize resources of an application and/or play a game.

At 402, a first plurality of sets of event information associated with afirst plurality of events may be identified. In some examples, an eventof the first plurality of events may be associated with a presentationof a content item (e.g., a content item, such as an advertisement,provided by the content system) via a client device, wherein the contentitem may be presented via the client device in response to a request forcontent. In some examples, the first plurality of events comprises aplurality of intentional click events, a plurality of accidental clickevents and/or a plurality of skip events. The first plurality of sets ofevent information may comprise a second plurality of sets of eventinformation associated with the plurality of intentional click events, athird plurality of sets of event information associated with theplurality of accidental click events and/or a fourth plurality of setsof event information associated with the plurality of skip events.

In some examples, an intentional click event of the plurality ofintentional click events may correspond to an event in which a contentitem (e.g., a content item provided by the content system) is presentedvia a client device and a selection (e.g., a click) of the content itemis received via the client device, where the selection is determined(e.g., predicted) to be intentional. In some examples, the selection maybe determined (e.g., predicted) to be intentional based upon a dwelltime associated with the intentional click event (such as using one ormore of the techniques discussed below).

In some examples, an accidental click event of the plurality ofaccidental click events may correspond to an event in which a contentitem (e.g., a content item provided by the content system) is presentedvia a client device and a selection (e.g., a click) of the content itemis received via the client device, where the selection is determined(e.g., predicted) to be accidental. In some examples, the selection maybe determined (e.g., predicted) to be accidental based upon a dwell timeassociated with the accidental click event (such as using one or more ofthe techniques discussed below).

In some examples, a skip event of the plurality of skip events maycorrespond to an event in which a content item (e.g., a content itemprovided by the content system) is presented via a client device and aselection (e.g., a click) of the content item is not received via theclient device (e.g., the content item is not selected and/or clickedwhile the content item is presented via the client device).

In some examples, a set of event information of the first plurality ofsets of event information may comprise information associated with anevent of the first plurality of events. In some examples, the set ofevent information may comprise content item information associated witha content item associated with the event (e.g., the content item maycorrespond to a content item that is presented and/or selected in theevent), client information associated with a client device and/or a userassociated with the event (e.g., the client device may correspond to aclient device that receives, selects and/or presents the content item,and/or the user may correspond to a user of the client device), and/orinternet resource information associated with an internet resourceassociated with the event (e.g., the internet resource may correspond toan internet resource on which the content item is presented in theevent). The internet resource may be at least one of a web page, awebsite, an application (e.g., a client application, a mobileapplication, a platform, etc.).

In an example, the content item information may be indicative of atleast one of the content item associated with the event, a content itemidentifier that identifies the content item, a brand, advertiser and/orcompany associated with the content item, one or more topics of thecontent item, one or more products and/or services associated with thecontent item (e.g., the content item may be used to advertise and/orpromote the one or more products and/or the one or more services), aformat of the content item (indicative of whether the content item isaudio, video or an image, for example), a duration and/or size of thecontent item, etc.

In an example, the client information may be indicative of at least oneof the client device, a device identifier associated with the clientdevice, an IP address associated with the client device, a media accesscontrol (MAC) address associated with the client device, a carrieridentifier indicative of carrier information associated with the clientdevice, a user identifier (e.g., at least one of a username, an emailaddress, a user account identifier, etc.) associated with the clientdevice and/or the user, a browser cookie (and/or a cookie identifierassociated with the client device), activity information (e.g., searchhistory information, website browsing history, email information, etc.)associated with the client device, the user identifier and/or the user,user demographic information (e.g., age, gender, etc.) associated withthe client device, the user identifier and/or the user, locationinformation associated with the client device, the user identifierand/or the user, etc. In some examples, the client information may bedetermined based upon information received from the client device(and/or one or more other devices associated with the user and/or a useraccount associated with the user). Alternatively and/or additionally,the client information may be generated based upon information receivedfrom servers associated with internet resources (e.g., at least one ofweb pages, applications, mobile applications, etc.) accessed and/orvisited by the client device and/or the user.

In an example, the internet resource information may be indicative of atleast one of the internet resource, an internet resource identifierassociated with the internet resource, a host device associated with theinternet resource (e.g., the host device may comprise one or morecomputing devices, storage and/or a network configured to host theinternet resource), a host identifier of the host device, a domain(e.g., a domain name, a top-level domain, etc.) associated with theinternet resource, an application identifier associated with theinternet resource (e.g., an application), a publisher identifierassociated with a publisher of the internet resource, etc.

A first set of event information of the first plurality of sets of eventinformation may be associated with a first event of the first pluralityof events. The first event may be associated with a first content item,a first client device (and/or a first user associated with the firstclient device) and/or a first internet resource (e.g., a web page, anapplication, a mobile application, etc.).

FIGS. 5A-5M illustrate examples of a system 501 for determining clickprobabilities associated with content items and/or selecting content fortransmission to devices, described with respect to the method 400 ofFIG. 4 . FIGS. 5A-5G illustrate examples of the first event associatedwith the first content item (shown with reference number 546 in FIG.5E), the first client device (shown with reference number 500 in FIG.5A), and/or the first internet resource (e.g., a fourth web page 544illustrated in FIG. 5E). The first user (and/or the first client device500) may access and/or interact with a service, such as a browser,software, a website, an application, an operating system, an emailinterface, a messaging interface, a music-streaming application, a videoapplication, etc. that provides a platform for accessing internetresources and/or viewing and/or downloading content from a serverassociated with the content system. In some examples, the content systemmay use a first user profile associated with the first client device 500and/or the first user to select content for presentation to the firstuser. In some examples, the first user profile may comprise at least oneof first activity information (e.g., activity information associatedwith at least one of the first client device 500, the first user, afirst user identifier associated with the first client device 500 and/orthe first user, etc.), first user demographic information (e.g., userdemographic information associated with at least one of the first clientdevice 500, the first user, the first user identifier, etc.), firstlocation information (e.g., location information associated with atleast one of the first client device 500, the first user, the first useridentifier, etc.), etc.

FIG. 5A illustrates the first client device 500 presenting and/oraccessing a first web page 508 using a browser of the first clientdevice 500. The browser may comprise an address bar 502 comprising a webaddress (e.g., a uniform resource locator (URL)) of the first web page508. The first web page 508 may comprise a search interface. Forexample, the search interface may comprise a web search engine designedto search for information throughout the internet. In some examples, thefirst web page 508 may comprise a search field 506. For example, a query“stock market” may be entered into the search field 506. In someexamples, the first web page 508 may comprise a search selectable input504 corresponding to performing a search based upon the query. Forexample, the search selectable input 504 may be selected.

FIG. 5B illustrates the first client device 500 presenting a pluralityof search results associated with the query using the browser of thefirst client device 500. For example, the plurality of search resultsmay be presented within a second web page 518. For example, theplurality of search results may comprise a first search result 510corresponding to a third web page, a second search result 512corresponding to the fourth web page 544 (illustrated in FIG. 5E), athird search result 514 corresponding to a fifth web page and/or afourth search result 516 corresponding to a sixth web page.

In some examples, each search result of the plurality of search resultsmay comprise a selectable input (e.g., a link) corresponding toaccessing a web page associated with the search result. In someexamples, the second search result 512 corresponding to the fourth webpage 544 may be selected (e.g., the second search result 512 may beselected via a second selectable input corresponding to the secondsearch result 512).

FIG. 5C illustrates the first client device 500 transmitting a request522 to access a resource to a first server 524. In some examples, therequest 522 to access the resource may be transmitted in response to thesecond search result 512 being selected. For example, the resource maycorrespond to the fourth web page 544. For example, the request 522 toaccess the resource may comprise an indication of the fourth web page544 (e.g., a web address “https://stocks.exchange.com”). Alternativelyand/or additionally, the first server 524 may be associated with thefourth web page 544.

FIG. 5D illustrates the first server 524 transmitting a first requestfor content 536 to a second server 538 associated with the contentsystem. In some examples, the first request for content 536 may betransmitted (by the first server 524) in response to receiving therequest 522 to access the resource. Alternatively and/or additionally,the first request for content 536 may be transmitted (to the secondserver 538) by the first client device 500. For example, in response tothe first server 524 receiving the request 522 to access the resource,the first server 524 (associated with the fourth web page 544, forexample) may transmit first resource information associated with thefourth web page 544 to the first client device. The first client device500 may transmit the first request for content 536 to the second server538 in response to receiving the first resource information. In someexamples, the first request for content 536 may be a request to beprovided with a content item (e.g., an advertisement, an image, a link,a video, etc.) for presentation via the fourth web page 544.

In some examples, the first request for content 536 may compriseidentification information associated with the first client device 500,the first user and/or the first internet resource (e.g., the fourth webpage 544). For example, the identification information may be used toidentify the first internet resource (e.g., the fourth web page 544)and/or the first user profile associated with the first client device500.

In some examples, a bidding process associated with the first requestfor content 536 may be performed to select a content item from a firstplurality of content items participating in an auction (e.g., an auctionfor selection of a content item to present via the first client device500). In some examples, the first plurality of content items(participating in the auction) may comprise the first content item 546.

In some examples, a first bid value associated with the first contentitem 546 may be determined. The first bid value may be determined basedupon at least one of a budget associated with the first content item546, a first target audience associated with the first content item 546,one or more advertisement campaign goals associated with the firstcontent item 546, a first content item bid value associated with thefirst content item 546, etc.

Alternatively and/or additionally, the first bid value may be determinedbased upon a first click probability associated with the first contentitem 546. In some examples, the first click probability isrepresentative of (e.g., comprises) a probability of receiving aselection (e.g., a click) of the first content item 546 in response topresenting the first content item 546 via the first client device 500(e.g., a probability that presentation of the first content item 546 viathe first client device 500 would be followed by a selection, such as aclick, of the first content item 546 on the first client device 500). Insome examples, the first click probability is determined based upon thefirst user profile, internet resource information associated with thefirst internet resource (e.g., the fourth web page 544), and/or contentitem information associated with the first content item 546. In anexample, the internet resource information may be indicative of at leastone of the first internet resource, a first internet resource identifierassociated with the first internet resource, a first host deviceassociated with the first internet resource, a first host identifier ofthe host device, a first domain (e.g., a domain name, a top-leveldomain, etc.) associated with the first internet resource, a firstapplication identifier associated with the first internet resource(e.g., an application), a first publisher identifier associated with apublisher of the first internet resource, etc. Alternatively and/oradditionally, the content item information may be indicative of at leastone of the first content item 546, a first content item identifier thatidentifies the first content item 546, a first brand, advertiser and/orcompany associated with the first content item 546, one or more firsttopics of the first content item 546, one or more first products and/orservices associated with the first content item 546 (e.g., the firstcontent item 546 may be used to advertise and/or promote the one or morefirst products and/or services), a first format of the first contentitem 546, a first duration and/or first size of the first content item546, etc. In some examples, the first click probability may bedetermined using one or more of the techniques discussed below withrespect to determining a second click probability.

The first bid value may correspond to a value of presenting the firstcontent item 546 via the first client device 500, such as determinedbased upon at least one of the first click probability, an amount ofrevenue (indicated by the first content item bid value, for example)associated with receiving a selection of the first content item 546 viathe first client device 500, etc.

In some examples, a first plurality of bid values (comprising the firstbid value) associated with the first plurality of content items(participating in the auction) may be compared to identify a winner ofthe auction. In some examples, the winner may correspond to a contentitem, of the first plurality of content items, associated with a highestbid value among the first plurality of bid values. For example, thefirst content item 546 may be selected for presentation via the firstclient device 500 based upon a determination that the first bid value isthe highest bid value among the first plurality of bid values (and/or adetermination that the first content item 546 is the winner of theauction).

In some examples, in response to selecting the first content item 546for presentation via the first client device 500, the first content item546 may be transmitted to the first client device 500 for presentationvia the fourth web page 544. FIG. 5E illustrates the first client device500 presenting and/or accessing the fourth web page 544 using thebrowser. For example, the content system may provide the first contentitem 546 to be presented via the fourth web page 544 while the fourthweb page 544 is accessed by the first client device 500.

In some examples, the first event may be determined to be a skip event(and may be included in the plurality of skip events, for example) basedupon a determination that the first content item 546 is not selected(e.g., clicked) via the first client device 500 while the first contentitem 546 is presented via the first client device 500. In an example,occurrence of the skip event may be detected based upon a determinationthat the content system does not receive information indicative ofoccurrence of a selection of the first content item 546, via the firstclient device 500, while the first content item 546 is presented via thefirst client device 500.

FIGS. 5F-5G illustrate an example scenario in which the first event isan intentional click event or an accidental click event. In someexamples, a selection of the first content item 546 may be received. Inresponse to the selection of the first content item 546, the firstclient device 500 may present and/or access a second internet resource(e.g., a seventh web page 552) associated with the first content item546, such as illustrated in FIG. 5F. In an example in which the firstcontent item 546 is an advertisement, the second internet resource(e.g., the seventh web page 552) may correspond to a landing page of theadvertisement, such as a web page comprising information associated withone or more products and/or one or more services promoted by theadvertisement. In some examples, after the first client device 500presents and/or access the second internet resource (e.g., the seventhweb page 552), the first client device 500 may leave the first internetresource (e.g., the fourth web page 544), such as by at least one ofclosing the browser, navigating to a different internet resource (e.g.,a different web page), returning to the first internet resource (e.g.,the fourth web page 544) such as in response to a selection of a backselectable input 548, etc. For example, in response to a selection ofthe back selectable input 548, the first client device 548 may return tothe first internet resource (e.g., the fourth web page 544), such asillustrated in FIG. 5G.

In some examples, the first event may be determined to be an accidentalclick event (and may be included in the plurality of accidental clickevents, for example) based upon a determination that a dwell timeassociated with the selection of the first content item 546 is less thana threshold dwell time (e.g., 3 seconds or other duration of time). Inan example, occurrence of the accidental click event may be detectedbased upon a determination that the dwell time is less than thethreshold dwell time. In some examples, the dwell time may be determinedbased upon a first time and a second time. For example, the dwell timemay be determined based upon a difference between the first time and thesecond time (e.g., if the first time is 9:00:00 and the second time is9:00:02, the dwell time may be determined to be 2 seconds). In someexamples, the first time may correspond to at least one of a time atwhich the first content item 546 is selected via the first client device500, a time at which the second internet resource (e.g., the seventh webpage 552) is accessed in response to the selection of the first contentitem 546, a time at which the first client device 500 leaves the firstinternet resource (e.g., the fourth web page 544) in response to theselection of the first content item 546, etc. In some examples, thesecond time may correspond to at least one of a time at which the firstclient device 500 returns to the first internet resource (e.g., thefourth web page 544) after the first content item 546 is selected (inresponse to the selection of the back selectable input 548, forexample), a time at which the first client device 500 leaves the secondinternet resource (such as by at least one of closing the browser,navigating to a different internet resource (e.g., a different webpage), returning to the first internet resource, etc.), etc.

In some examples, the first event may be determined to be an intentionalclick event (and may be included in the plurality of intentional clickevents, for example) based upon a determination that the dwell timeassociated with the selection of the first content item 546 exceeds thethreshold dwell time (e.g., the dwell time may be determined based uponthe first time and/or the second time). In an example, occurrence of theintentional click event may be detected based upon a determination thatthe dwell time exceeds the threshold dwell time. Alternatively and/oradditionally, occurrence of the intentional click event may be detectedbased upon a determination that the first content item 546 is selectedvia the first client device 500 (while the first content item 546 ispresented via the first internet resource) and/or that the first clientdevice 500 does not return to the first internet resource (e.g., thefourth web page 544) from the second internet resource (e.g., theseventh web page 552). Alternatively and/or additionally, occurrence ofthe intentional click event may be detected based upon a determinationthat the first content item 546 is selected via the first client device500 (while the first content item 546 is presented via the firstinternet resource) and/or that the first client device 500 does notaccess the first internet resource (and/or another internet resourceother than the second internet resource) for a threshold period of timeafter the first time.

In some examples, the threshold dwell time may be a first global dwelltime threshold applied for determining whether events of the firstplurality of events are accidental click events or intentional clickevents. Alternatively and/or additionally, the threshold dwell time maybe a first traffic segment threshold of a plurality of traffic segmentthresholds, wherein each traffic segment threshold of the plurality oftraffic segment threshold may be applied for determined whether eventsof a subset of the first plurality of events are accidental click eventsor intentional click events. For example, the first traffic segmentthreshold may be applied to determine whether the first event is anintentional click event or an accidental click event if the first eventis associated with one or more first features associated with the firsttraffic segment threshold. Alternatively and/or additionally, a secondtraffic segment threshold different than the first traffic segmentthreshold may be applied to determine whether a different event is anintentional click event or an accidental click event if the differentevent is associated with one or more second features associated with thesecond traffic segment threshold.

In an example, the one or more first features associated with the firsttraffic segment threshold may comprise at least one of one or more firstoperating systems, one or more first internet resources, one or morefirst types of devices, etc. Alternatively and/or additionally, the oneor more second features associated with the second traffic segment maycomprise at least one of one or more second operating systems, one ormore second internet resources, one or more second types of devices,etc. In an example, the first traffic segment threshold (rather than thesecond traffic segment threshold, for example) may be applied todetermine whether the first event is an intentional click event or anaccidental click event based upon at least one of a determination thatan operating system of the first client device 500 matches an operatingsystem of the one or more first operating systems, a determination thatthe first internet resource matches an internet resource of the one ormore first internet resources, a determination that the second internetresource matches an internet resource of the one or more first internetresources, a determination that a type of device of the first clientdevice 500 matches a type of the device of the one or more first typesof devices, etc.

In some examples, the first set of event information associated with thefirst event may be indicative of first content item informationassociated with the first content item 546 of the first event, firstclient information associated with the first client device 500 (and/orthe first user) of the first event, and/or first internet resourceinformation associated with the first internet resource (e.g., thefourth web page 544) of the first event.

In an example, the first content item information may be indicative ofat least one of the first content item 546, the first content itemidentifier that identifies the first content item 546, the first brand,advertiser and/or company associated with the first content item 546,the one or more first topics of the first content item 546, the one ormore first products and/or services associated with the first contentitem 546, the first format of the first content item 546, the firstduration and/or first size of the first content item 546, etc.

In an example, the first client information may be indicative of atleast one of the first client device 500, a first device identifierassociated with the first client device 500, a first IP addressassociated with the first client device 500, a first MAC addressassociated with the first client device 500, a first carrier identifierindicative of carrier information associated with the first clientdevice 500, the first user identifier (e.g., at least one of a username,an email address, a user account identifier, etc.) associated with thefirst client device 500 and/or the first user, a first browser cookie(and/or a first cookie identifier associated with the first clientdevice 500), the first activity information (e.g., search historyinformation, website browsing history, email information, etc.), thefirst user demographic information (e.g., age, gender, etc.), the firstlocation information, etc. In some examples, at least some of the firstclient information may be determined based upon the first user profileassociated with the first client device 500 and/or the first user.

In an example, the first internet resource information may be indicativeof at least one of the first internet resource, the first internetresource identifier associated with the first internet resource, thefirst host device associated with the first internet resource, the firsthost identifier of the host device, the first domain (e.g., a domainname, a top-level domain, etc.) associated with the first internetresource, the first application identifier associated with the firstinternet resource (e.g., an application), the first publisher identifierassociated with a publisher of the first internet resource, etc.

At 404, a plurality of accidental click probabilities associated with asecond plurality of events of the first plurality of events may bedetermined. In some examples, the second plurality of events (associatedwith the plurality of accidental click probabilities) may comprise theplurality of accidental click events and/or the plurality of skip eventsthe plurality of intentional click events. In an example, the secondplurality of events may merely comprise the plurality of accidentalclick events and the plurality of skip events (without any intentionalclick event of the plurality of intentional click events, for example).Alternatively and/or additionally, the second plurality of events maymerely comprise the plurality of skip events (without any intentionalclick event of the plurality of intentional click events and without anyaccidental click event of the plurality of accidental click events, forexample). Alternatively and/or additionally, the second plurality ofevents may merely comprise the plurality of accidental click events(without any intentional click event of the plurality of intentionalclick events and without any skip event of the plurality of skip events,for example).

In some examples, the plurality of accidental click probabilities may bedetermined based upon sets of event information, of the first pluralityof sets of event information, associated with the second plurality ofevents. For example, in a scenario in which the second plurality ofevents comprises the plurality of accidental click events, accidentalclick probabilities associated with the plurality of accidental clickevents may be determined based upon the third plurality of sets of eventinformation associated with the plurality of accidental click events.Alternatively and/or additionally, in a scenario in which the secondplurality of events comprises the plurality of skip events, accidentalclick probabilities associated with the plurality of skip events may bedetermined based upon the fourth plurality of sets of event informationassociated with the plurality of skip events. Alternatively and/oradditionally, in a scenario in which the second plurality of eventscomprises the plurality of intentional click events, accidental clickprobabilities associated with the plurality of intentional click eventsmay be determined based upon the second plurality of sets of eventinformation associated with the plurality of intentional click events.

In an example, an accidental click probability of the plurality ofaccidental click probabilities may be associated with an event (of thesecond plurality of events) associated with presentation of a contentitem via a client device. The accidental click probability may bedetermined based upon a set of event information, of the first pluralityof sets of event information, associated with the event. The accidentalclick probability may be representative of (e.g., comprises) aprobability that presentation of the content item via the client devicewould be followed by occurrence of an accidental click event (in whichthe content item is accidentally selected and/or a dwell time associatedwith a selection of the content item is less than the threshold dwelltime, for example). For example, although it may be known whether or notthe event is an accidental click event when the accidental clickprobability is determined, the accidental click probability may bedetermined, based upon the set of event information associated with theevent, without considering whether or not the event is determined to bean accidental click event. Alternatively and/or additionally, whether ornot the event is an accidental click event may be considered indetermining the accidental click probability.

In some examples, in a scenario in which the second plurality of eventscomprises accidental click events, one or more accidental clickprobabilities associated with one or more accidental click events may bebetween 0 and 1 (and/or between 0% and 100%). Alternatively and/oradditionally, in a scenario in which the second plurality of eventscomprises non-accidental click events (e.g., skip events and/orintentional click events), one or more accidental click probabilitiesassociated with one or more non-accidental click events may be between 0and 1 (and/or between 0% and 100%).

In an example in which the first event is included in the secondplurality of events, a first accidental click probability, associatedwith the first event, may be determined based upon the first set ofevent information associated with the first event. The first accidentalclick probability associated with the first event may be representativeof (e.g., comprises) a probability that, given the first set of eventinformation associated with the first event, presentation of the firstcontent item 546 via the first client device 500 would be followed byoccurrence of an accidental click event (e.g., a probability thatpresenting the first content item 546 in response to the first requestfor content 536 is followed by an accidental selection of the firstcontent item 546). In some examples, whether or not the first event isan accidental click event may not be considered in determining the firstaccidental click probability. For example, even if the first event is anaccidental click event, the first accidental click probabilityassociated with the first event may be determined to be a probabilityother than 1 (and/or 100%) (e.g., the first accidental click probabilityassociated with the first event may be determined to be between 0 and 1and/or between 0% and 100%). Alternatively and/or additionally, even ifthe first event is a non-accidental click event (e.g., a skip eventand/or an intentional click event), the first accidental clickprobability associated with the first event may be determined to be aprobability other than 0 (and/or 0%) (e.g., the first accidental clickprobability associated with the first event may be determined to bebetween 0 and 1 and/or between 0% and 100%). Alternatively and/oradditionally, whether or not the first event is an accidental clickevent may be considered in determining the first accidental clickprobability. For example, the first accidental click probability may beincreased (by a value, by a proportion, by a percentage and/or by afactor, for example) in a scenario in which the first event isconsidered to be an accidental click event as compared to a scenario inwhich the first event is considered to be a non-accidental click event(such as a skip event and/or an intentional click event). In an example,in a scenario in which the first event is considered to be an accidentalclick event, the first accidental click probability may be determined tobe 0.06 (and/or 6%), and/or in a scenario in which the first event isconsidered to be a non-accidental click event, the first accidentalclick probability may be determined to be 0.03 (and/or 3%).

In some examples, the first accidental click probability may bedetermined based upon a first proportion of events (e.g., eventscomprising accidental click events, skip events and/or intentional clickevents) that are accidental click events. For example, the firstproportion of events that are accidental click events may correspond toa proportion of events, of the first plurality of events, that areaccidental click events (such as determined based upon a quantity of thefirst plurality of events and/or a quantity of accidental click eventsof the plurality of accidental click events). In an example, theaccidental click probability may be equal to the first proportion ofevents that are accidental click events. Alternatively and/oradditionally, one or more operations (e.g., mathematical operations) maybe performed using the first proportion of events that are accidentalclick events (and/or one or more other values) to determine the firstaccidental click probability.

Alternatively and/or additionally, the first accidental clickprobability may be determined based upon the first internet resource(e.g., the fourth web page 544) associated with the first event. Forexample, a second proportion of events (e.g., events comprisingaccidental click events, skip events and/or intentional click events)that are accidental click events may be determined based upon the firstinternet resource. For example, the second proportion of events that areaccidental click events may correspond to a proportion of events, ofevents associated with the first internet resource, that are accidentalclick events (such as determined based upon a quantity of the eventsassociated with the first internet resource and/or a quantity ofaccidental click events associated with the first internet resource). Inan example, the accidental click probability may be equal to the secondproportion of events that are accidental click events. Alternativelyand/or additionally, one or more operations (e.g., mathematicaloperations) may be performed using the second proportion of events thatare accidental click events (and/or one or more other values, such asthe first proportion of events that are accidental click events) todetermine the first accidental click probability.

Alternatively and/or additionally, the first accidental clickprobability may be determined based upon the first content item 546associated with the first event. For example, a third proportion ofevents (e.g., events comprising accidental click events, skip eventsand/or intentional click events) that are accidental click events may bedetermined based upon the first content item 546. For example, the thirdproportion of events that are accidental click events may correspond toa proportion of events, of events associated with the first content item546, that are accidental click events (such as determined based upon aquantity of the events associated with the first content item 546 and/ora quantity of accidental click events associated with the first contentitem 546). In an example, the accidental click probability may be equalto the third proportion of events that are accidental click events.Alternatively and/or additionally, one or more operations (e.g.,mathematical operations) may be performed using the third proportion ofevents that are accidental click events (and/or one or more othervalues, such as the first proportion of events that are accidental clickevents and/or the second proportion of events that are accidental clickevents) to determine the first accidental click probability.

In some examples, accidental click probabilities, of the plurality ofaccidental click probabilities, other than the first accidental clickprobability may be determined using one or more of the techniquesdiscussed herein with respect to determining the first accidental clickprobability.

Alternatively and/or additionally, the plurality of accidental clickprobabilities may be determined using a first machine learning model. Insome examples, machine learning model training may be performed usingfirst training data to generate the first machine learning model. FIG.5H illustrates a machine learning training module 562 performing machinelearning model training to generate the first machine learning model(shown with reference number 564 in FIG. 5H). In an example, the firsttraining data (shown with reference number 560 in FIG. 5H) may be inputto the machine learning training module 562. The machine learningtraining module 562 may generate the first machine learning model 564using the first training data 560.

In some examples, the first training data 560 may comprise accidentalclick event information associated with a second plurality of accidentalclick events and/or skip event information associated with a secondplurality of skip events. In some examples, the second plurality ofaccidental click events may be the same as the plurality of accidentalclick events of the first plurality of events. Alternatively and/oradditionally, the second plurality of accidental click events may bedifferent than the plurality of accidental click events. Alternativelyand/or additionally, the second plurality of accidental click events maycomprise one, some and/or all of the plurality of accidental clickevents. Alternatively and/or additionally, the second plurality ofaccidental click events may not comprise any of the plurality ofaccidental click events.

In some examples, the second plurality of skip events may be the same asthe plurality of skip events of the first plurality of events.Alternatively and/or additionally, the second plurality of skip eventsmay be different than the plurality of skip events. Alternatively and/oradditionally, the second plurality of skip events may comprise one, someand/or all of the plurality of skip events. Alternatively and/oradditionally, the second plurality of skip events may not comprise anyof the plurality of skip events.

In some examples, the accidental click information may comprise a fifthplurality of sets of event information associated with the secondplurality of accidental click events, wherein a set of event informationof the fifth plurality of sets of event information may comprise atleast one of content item information, client information, internetresource information, etc. associated with an accidental click event ofthe second plurality of sets of accidental click events. In someexamples, the fifth plurality of sets of event information may be thesame as the third plurality of sets of event information of the firstplurality of sets of event information. Alternatively and/oradditionally, the fifth plurality of sets of event information may bedifferent than the third plurality of sets of event information.Alternatively and/or additionally, the fifth plurality of sets of eventinformation may comprise one, some and/or all of the third plurality ofsets of event information. Alternatively and/or additionally, the fifthplurality of sets of event information may not comprise any of the thirdplurality of sets of event information.

In some examples, the skip information may comprise a sixth plurality ofsets of event information associated with the second plurality of skipevents, wherein a set of event information of the sixth plurality ofsets of event information may comprise at least one of content iteminformation, client information, internet resource information, etc.associated with a skip event of the second plurality of sets of skipevents. In some examples, the sixth plurality of sets of eventinformation may be the same as the fourth plurality of sets of eventinformation of the first plurality of sets of event information.Alternatively and/or additionally, the sixth plurality of sets of eventinformation may be different than the fourth plurality of sets of eventinformation. Alternatively and/or additionally, the sixth plurality ofsets of event information may comprise one, some and/or all of thefourth plurality of sets of event information. Alternatively and/oradditionally, the sixth plurality of sets of event information may notcomprise any of the fourth plurality of sets of event information.

FIG. 5I illustrates an example of the first training data 560. In someexamples, the first training data 560 may comprise a plurality of setsof training data 568 and/or first target information 570 (e.g., targetattributes associated with the first plurality of sets of trainingdata). The plurality of sets of training data 568 may comprise the fifthplurality of sets of event information and/or the sixth plurality ofsets of event information. For example, a first subset of the pluralityof sets of training data 568 may comprise the fifth plurality of sets ofevent information and/or a second subset of the plurality of sets oftraining data 568 may comprise the sixth plurality of sets of eventinformation (e.g., each set of training data of the first subset maycomprise a set of event information of the fifth plurality of sets ofevent information and/or each set of training data of the second subsetmay comprise a set of event information of the sixth plurality of setsof event information).

In some examples, the first target information 570 may comprise a firstplurality of labels. The first plurality of labels comprises a secondplurality of labels associated with the fifth plurality of sets of eventinformation (e.g., the first subset) and/or a third plurality of labelsassociated with the sixth plurality of sets of event information (e.g.,the second subset). In some examples, the first plurality of labels maybe indicative of classifications. In some examples, the second pluralityof labels, that are associated with the fifth plurality of sets of eventinformation, may be indicative of a first classification (e.g., aclassification corresponding to accidental click events) and/or a firstvalue (e.g., 1). Alternatively and/or additionally, the third pluralityof labels, that are associated with the sixth plurality of sets of eventinformation, may be indicative of a second classification (e.g., aclassification corresponding to skip events) and/or a second value(e.g., 0).

In FIG. 5I, a label of the second plurality of labels (shown as“ACCIDENTAL (1)” in FIG. 5I) is associated with a set of eventinformation (shown as “AC EVENT INFO” in FIG. 5I) of the fifth pluralityof sets of event information associated with the second plurality ofaccidental click events. A label of the third plurality of labels (shownas “SKIP (0)” in FIG. 5I) is associated with a set of event information(shown as “SKIP EVENT INFO” in FIG. 5I) of the sixth plurality of setsof event information associated with the second plurality of skipevents. In some examples, the fifth plurality of sets of eventinformation associated with the second plurality of accidental clickevents may be labeled as corresponding to positive events. Alternativelyand/or additionally, the sixth plurality of sets of event informationassociated with the second plurality of skip events may be labeled ascorresponding to negative events. For example, a value (e.g., 0)indicated by the third plurality of labels may be lower than a value(e.g., 1) indicated by the second plurality of labels.

In some examples, the first machine learning model 564 may be trainedand/or configured to determine an accidental click probability basedupon a set of event information, such as a set of event informationassociated with an event of the second plurality of events. FIG. 5Jillustrates the plurality of accidental click probabilities (shown withreference number 578 in FIG. 5J) being determined using the firstmachine learning model 564. In an example, the first machine learningmodel 564 may be loaded into an accidental click probability predictionmodule 576. A plurality of sets of event information 574 associated withthe second plurality of events may be input to the accidental clickprobability prediction module 576. The accidental click probabilityprediction module 576 may use the first machine learning model 564 todetermine the plurality of accidental click probabilities 578 based uponthe plurality of sets of event information 574. For example, anaccidental click probability, of the plurality of accidental clickprobabilities 578, associated with an event may be determined (using theaccidental click probability prediction module 576 and/or the firstmachine learning model 564, for example) based upon a set of eventinformation, of the plurality of sets of event information 574,associated with the event. Alternatively and/or additionally, for eachevent of the second plurality of events, an accidental click probability(of the plurality of accidental click probabilities 578) may bedetermined (using the accidental click probability prediction module 576and/or the first machine learning model 564, for example) based upon aset of event information, of the plurality of sets of event information574, associated with the event.

At 406, machine learning model training may be performed using the firstplurality of sets of event information and a fourth plurality of labelsassociated with the first plurality of events to generate a secondmachine learning model. FIG. 5K illustrates the machine learningtraining module 562 performing machine learning model training togenerate the second machine learning model (shown with reference number582 in FIG. 5K). In an example, second training data 580 may be input tothe machine learning training module 562. The machine learning trainingmodule 562 may generate the second machine learning model 582 using thesecond training data 582.

In some examples, the second training data 580 may comprise the firstplurality of sets of event information and/or the fourth plurality oflabels. FIG. 5L illustrates an example of the second training data 580.In some examples, the second training data 580 may comprise the firstplurality of sets of event information (shown with reference number 586in FIG. 5L) and/or second target information 588 (e.g., targetattributes associated with the first plurality of sets of eventinformation 586). In some examples, the second target information 588comprises the fourth plurality of labels.

In some examples, the fourth plurality of labels comprises a fifthplurality of labels and a sixth plurality of labels. The fifth pluralityof labels may be associated with the plurality of intentional clickevents and/or the second plurality of sets of event information(associated with the plurality of intentional click events). In someexamples, labels of the fifth plurality of labels may correspond to anintentional click classification (e.g., a classification correspondingto intentional click events). For example, in FIG. 5L, labels of thefifth plurality of labels are shown as “INTENTIONAL (1)”. A label of thefifth plurality of labels may indicate that a set of event information(of the second plurality of sets of event information), associated withthe label, is associated with an intentional click event. In someexamples, the second plurality of sets of event information associatedwith the plurality of accidental click events may be labeled ascorresponding to positive events (e.g., 1). For example, the fifthplurality of labels may comprise positive labels (indicative of 1, forexample). In some examples, each label of the second plurality of labelsmay be indicative of a value (e.g., 1) associated with the intentionalclick classification.

The sixth plurality of labels may be associated with the secondplurality of events. The sixth plurality of labels may be based upon(e.g., may correspond to and/or be indicative of) the plurality ofaccidental click probabilities. For example, a label (of the sixthplurality of labels) that is associated with an event of the secondplurality of events may be determined based upon an accidental clickprobability, of the plurality of accidental click probabilities,associated with the event. In an example, the label may be indicative ofa value that is equal to the accidental click probability. Alternativelyand/or additionally, the label may be indicative of a value that isdifferent than the accidental click probability (e.g., one or moreoperations, such as mathematical operations, may be performed using theaccidental click probability and one or more other values to determinethe value of the label).

In an example in which the second plurality of events (for whichaccidental probabilities of the plurality of accidental probabilitiesare determined, for example) comprises the plurality of accidental clickevents, the sixth plurality of labels may comprise first labelsassociated with the plurality of accidental click events and/or thethird plurality of sets of event information (associated with theplurality of accidental click events). In some examples, the firstlabels may be based upon (e.g., may correspond to and/or be indicativeof) accidental click probabilities, of the plurality of accidental clickprobabilities, associated with the plurality of accidental click events.In an example, the first labels may comprise a label 591 (shown as “ACPROB (0.05)”) associated with a set of event information 590 (shown as“AC EVENT INFO 1”) that is associated with an accidental click event ofthe plurality of accidental click events (e.g., the set of eventinformation 590 comprises event information corresponding to theaccidental click event). The label 591 may be based upon (e.g., maycorrespond to and/or be indicative of) an accidental click probability(e.g., 0.05), of the plurality of accidental click probabilities,associated with the accidental click event.

In an example in which the second plurality of events (for whichaccidental probabilities of the plurality of accidental probabilitiesare determined, for example) comprises the plurality of skip events, thesixth plurality of labels may comprise second labels associated with theplurality of skip events and/or the fourth plurality of sets of eventinformation (associated with the plurality of skip events). In someexamples, the second labels may be based upon (e.g., may correspond toand/or be indicative of) accidental click probabilities, of theplurality of accidental click probabilities, associated with theplurality of skip events. In an example, the second labels may comprisea label 585 (shown as “AC PROB (0.12)”) associated with a set of eventinformation 584 (shown as “SKIP EVENT INFO 1”) that is associated with askip event of the plurality of skip events (e.g., the set of eventinformation 584 comprises event information corresponding to the skipevent). The label 585 may be based upon (e.g., may correspond to and/orbe indicative of) an accidental click probability (e.g., 0.05), of theplurality of accidental click probabilities, associated with the skipevent.

Accordingly, in some examples, the fourth plurality of labels maycorrespond to a non-binary label set, where at least some of the fourthplurality of labels (such as labels, of the fourth plurality of labels,that are based upon accidental click probabilities) are indicative ofvalues that are not equal to 0 or 1. In some examples, the secondmachine learning model 582 may be trained using a first loss function.The first loss function may be able to handle a non-binary label set,such as the fourth plurality of labels. In some examples, the first lossfunction may comprise a binary cross-entropy loss function (and/or adifferent type of loss function). In an example, the first loss functionmay comprise

′(p,

)=

ln(

÷p)+(1−

)ln((1−

)÷(1−p)), where

with presentation of a content item via a client device, such as inresponse to a request for content) and/or p corresponds to a predictionassociated with the event (e.g., the prediction may correspond to aclick probability, associated with presenting the content item via theclient device, determined using the second machine learning model 582 inresponse to receiving the request for content). In an example, label

associated with the event may be determined after occurrence of theevent. In some examples, if the event is an intentional click event, thelabel

may correspond to the intentional click classification (e.g., the label

may be equal to 1). Alternatively and/or additionally, if the event isan accidental click event and/or a skip event, the label

may be based upon (e.g., may correspond to and/or be indicative of) anaccidental click probability associated with the event (e.g., anaccidental click probability determined using the accidental clickprobability prediction module 576 and/or the first machine learningmodel 564 using one or more of the techniques discussed herein). In someexamples, loss (e.g.,

′(p,

)) determined using the first loss function may be representative of adeviation of the prediction (e.g., the click probability determinedusing the second machine learning model 582) from the label.

In some examples, the second machine learning model 582 may be trainedand/or configured to determine a click probability associated withpresentation of a content item via a client device. The clickprobability may be determined in response to receiving a request forcontent associated with the client device. Alternatively and/oradditionally, the click probability may be determined based upon contentitem information associated with the content item, client informationassociated with the client device, and/or internet resource informationassociated with an internet resource associated with the request forcontent, etc.

At 408, a second request for content associated with a second clientdevice may be received. In some examples, the second request for contentmay be associated with a second internet resource (e.g., a web page, anapplication, a mobile application, etc.). For example, the secondrequest for content may be a request to be provided with a content item(e.g., an advertisement, an image, a link, a video, etc.) forpresentation via the second client device on the second internetresource.

In some examples, the second request for content may comprise secondidentification information associated with the second client device, asecond user associated with the second client device and/or the secondinternet resource. For example, the second identification informationmay be used to identify the second internet resource and/or a seconduser profile associated with the second client device.

At 410, a second plurality of click probabilities associated with asecond plurality of content items may be determined using the secondmachine learning model. In some examples, the second plurality of clickprobabilities may be determined in response to receiving the secondrequest for content. The second plurality of click probabilities maycomprise a second click probability associated with a second contentitem (e.g., an advertisement, an image, a link, a video, etc.) of thesecond plurality of content items. In some examples, the second clickprobability is representative of (e.g., comprises) a probability ofreceiving a selection (e.g., a click) of the second content itemresponsive to presenting the second content item via the second clientdevice (e.g., a probability that presentation of the second content itemvia the second client device would be followed by a selection, such as aclick, of the second content item on the second client device).

In some examples, one or more features associated with the secondrequest for content may be determined. The one or more features maycomprise second client information associated with the second clientdevice and/or second internet resource information associated with thesecond internet resource. Determining the second plurality of clickprobabilities using the second machine learning model may be based uponthe one or more features. In some examples, the second click probabilityis determined based upon the second client information associated withthe second client device, second content item information associatedwith the second content item and/or the second internet resourceinformation associated with the second internet resource.

In an example, the second content item information may be indicative ofat least one of the second content item, a second content itemidentifier that identifies the second content item, a second brand,advertiser and/or company associated with the second content item, oneor more second topics of the second content item, one or more secondproducts and/or services associated with the second content item (e.g.,the content item may be used to advertise and/or promote the one or moreproducts and/or the one or more services), a second format of the secondcontent item (indicative of whether the second content item is audio,video or an image, for example), a second duration and/or size of thesecond content item, etc.

In an example, the second client information may be indicative of atleast one of the second client device, a second device identifierassociated with the second client device, a second IP address associatedwith the second client device, a second MAC address associated with thesecond client device, a second carrier identifier indicative of secondcarrier information associated with the second client device, a seconduser identifier (e.g., at least one of a username, an email address, auser account identifier, etc.) associated with the second client deviceand/or the second user, a second browser cookie (and/or a second cookieidentifier associated with the second client device), second activityinformation (e.g., search history information, website browsing history,email information, etc.) associated with the second client device, thesecond user identifier and/or the second user, second user demographicinformation (e.g., age, gender, etc.) associated with the second clientdevice, the second user identifier and/or the second user, secondlocation information associated with the second client device, thesecond user identifier and/or the second user, etc.

In an example, the second internet resource information may beindicative of at least one of the second internet resource, a secondinternet resource identifier associated with the second internetresource, a second host device associated with the second internetresource (e.g., the second host device may comprise one or morecomputing devices, storage and/or a network configured to host thesecond internet resource), a second host identifier of the second hostdevice, a second domain (e.g., a domain name, a top-level domain, etc.)associated with the second internet resource, a second applicationidentifier associated with the second internet resource (e.g., anapplication), a second publisher identifier associated with a secondpublisher of the second internet resource, etc.

FIG. 5M illustrates the second click probability (shown with referencenumber 598 in FIG. 5M) being determined using the second machinelearning model 596. In an example, the second machine learning model 596may be loaded into a click probability prediction module 594.Information 592, comprising the second content item information, thesecond client information and/or the second internet resourceinformation, may be input to the click probability prediction module594. The click probability prediction module 594 may use the secondmachine learning model 596 to determine the second click probability 598based upon the information 592.

Alternatively and/or additionally, embodiments of the present disclosureare contemplated in which the second click probability (and/or otherclick probabilities of the second plurality of click probabilities) isdetermined using one or more techniques other than (and/or in additionto) using the second machine learning model 596. For example, the secondclick probability may be determined, based upon the information 592, theplurality of accidental click probabilities and/or the first pluralityof sets of event information, using one or more techniques other than(and/or in addition to) using the second machine learning model 596.

In some examples, click probabilities, other than the second clickprobability, of the second plurality of click probabilities may bedetermined using one or more of the techniques discussed herein withrespect to determining the second click probability associated with thesecond content item.

At 412, the second content item may be selected for presentation via thesecond client device based upon the second plurality of clickprobabilities. For example, the second content item may be selected forpresentation via the second client device based upon a determinationthat the second click probability associated with the second contentitem is a highest click probability among the second plurality of clickprobabilities.

Alternatively and/or additionally, a bidding process of a second auctionassociated with the second request for content may be performed toselect a content item from the second plurality of content items forpresentation via the second client device. In some examples, a secondplurality of bid values associated with the second plurality of contentitems may be determined based upon the second plurality of clickprobabilities. For example, the second plurality of bid values maycomprise a second bid value associated with the second content item. Insome examples, the second bid value may be determined based upon thesecond click probability and at least one of a budget associated withthe second content item, a second target audience associated with thesecond content item, one or more advertisement campaign goals associatedwith the second content item, a second content item bid value associatedwith the second content item, etc.

In some examples, the second bid value may correspond to a value ofpresenting the second content item via the second client device, such asdetermined based upon at least one of the second click probability, anamount of revenue (indicated by the second content item bid value, forexample) associated with a selection of the second content item via thesecond client device, etc. In an example where the second clickprobability is 10% and/or the amount of revenue associated withreceiving a selection of the second content item via the second clientdevice is $50.00, the second bid value may correspond to a combinationof the second click probability and the amount of revenue (e.g., thesecond bid value may correspond to 10% x $50.00=$5.00).

In some examples, the second plurality of bid values (comprising thesecond bid value) associated with the second plurality of content itemsmay be compared to identify a winner of the second auction. In someexamples, the winner may correspond to a content item, of the secondplurality of content items, associated with a highest bid value amongthe second plurality of bid values. For example, the second content itemmay be selected for presentation via the second client device based upona determination that the second bid value is the highest bid value amongthe second plurality of bid values (and/or a determination that thesecond content item is the winner of the second auction).

In some examples, in response to selecting the second content item forpresentation via the second client device, the second content item maybe transmitted to the second client device for presentation via thesecond internet resource. For example, the content system may providethe second content item to be presented via the second internet resourcewhile the second internet resource is accessed by the second clientdevice (e.g., the second client device may present the second contentitem via the second internet resource).

FIG. 6 illustrates an example of a system 601 for determining clickprobabilities associated with content items and/or selecting content fortransmission to devices, described with respect to the method 400 ofFIG. 4 . A logging system 612 may receive information 614 indicative ofuser actions from one or more client devices 628 and/or one or morecomputers (e.g., servers associated with internet resources accessed bythe one or more client devices 628). For example, the information 614may be indicative of user activity associated with events of the firstplurality of events. The logging system 612 may be configured to loginformation associated with the user activity and/or generate the firstplurality of sets of event information associated with the firstplurality of events. In some examples, the logging system 612 may outputtraining data 616 to a machine learning training system 602. The machinelearning training system 602 may be configured to use the training data616 to train an accidental click prediction model 604 (e.g., anaccidental click prediction auxiliary model), such as the first machinelearning model 564, and/or a click prediction model 618, such as thesecond machine learning model 582. In some examples, the training data616 may comprise the first plurality of sets of event information.Alternatively and/or additionally, the training data 616 may comprisethe first training data 560 for use in training the accidental clickprediction model 604. In some examples, the accidental click predictionmodel 604 may be used to generate accidental click predictions 608(indicative of the plurality of accidental click probabilities 578, forexample). The accidental click predictions 608 may be used to train theclick prediction model 618. In some examples, model data 620 of theclick prediction model 618 may be loaded into a content serving system624 (e.g., an advertisement serving system). In some examples, contentdata 610 (e.g., advertisement data) may be received by the contentserving system 624 from a content inventory 606 (e.g., an advertisementinventory), such as a data store that stores content items (e.g.,advertisements) and/or information (e.g., content item information)associated with the content items. In some examples, the content servingsystem 624 may use the click prediction model 618 to determine clickprobabilities associated with presenting content items via clientdevices. For example, the content serving system 624 may select contentitems for presentation via client devices based upon the clickprobabilities determined using the click prediction model 618. Thecontent serving system 624 may transmit content 622 (e.g., one or morerendered content items, such as one or more rendered advertisements) toone or more client devices 628, such as in response to selecting the oneor more content items for presentation via the one or more clientdevices 628.

Although various examples of the present disclosure are described withrespect to selection and/or presentation of content items comprisingadvertisements, embodiments are contemplated in which the content itemscomprise any type of content, such as at least one of search results,audio (e.g., songs, podcasts, etc.), video (e.g., movies, shows, videoclips, etc.), articles, social media feeds, suggested content (e.g.,links to videos, audio, articles, social media feeds, etc.), etc. whilestaying within the scope of the present disclosure.

Implementation of at least some of the disclosed subject matter may leadto benefits including, but not limited to, more accurate determinationof click probabilities associated with content items (e.g., as a resultof determining accidental click probabilities associated with the secondplurality of events, as a result of determining the click probabilitiesbased upon the accidental click probabilities, such as by way oftraining the second machine learning model 596 using the accidentalclick probabilities and/or determining the click probabilities using thesecond machine learning model 596, etc.). The more accuratedetermination of click probabilities may also lead to a more accurateand/or appropriate selection of a content item for presentation via aclient device that has a higher probability of resulting in the contentitem being selected and/or a higher probability of a user consuming thecontent item to have an interest in the content item.

Alternatively and/or additionally, implementation of at least some ofthe disclosed subject matter may lead to benefits including an increasein generalized revenue for presenting content items via client devices(e.g., as a result of the more accurate determination of clickprobabilities, etc.).

Alternatively and/or additionally, implementation of at least some ofthe disclosed subject matter may lead to a more accurate and/orappropriate selection of a content item for presentation via a clientdevice that has a higher probability of resulting in the content itembeing selected and/or a higher probability of a user consuming thecontent item to have an interest in the content item (e.g., as a resultof the more accurate determination of click probabilities, etc.).

Alternatively and/or additionally, implementation of at least some ofthe disclosed subject matter may lead to benefits including a reductionin screen space and/or an improved usability of a display (e.g., of theclient device) (e.g., as a result of the higher probability of the userconsuming the content item to have an interest in the content item,wherein the user may not view content that the user does not have aninterest in, wherein the user may not need to open a separateapplication and/or a separate window in order to find content having thesubject matter that the user has an interest in, etc.).

Alternatively and/or additionally, implementation of at least some ofthe disclosed subject matter may lead to benefits including a reductionin bandwidth (e.g., as a result of reducing a need for the user to opena separate application and/or a separate window in order to searchthroughout the internet and/or navigate through internet content to findcontent that the user has an interest in).

In some examples, at least some of the disclosed subject matter may beimplemented on a client device, and in some examples, at least some ofthe disclosed subject matter may be implemented on a server (e.g.,hosting a service accessible via a network, such as the Internet).

FIG. 7 is an illustration of a scenario 700 involving an examplenon-transitory machine readable medium 702. The non-transitory machinereadable medium 702 may comprise processor-executable instructions 712that when executed by a processor 716 cause performance (e.g., by theprocessor 716) of at least some of the provisions herein (e.g.,embodiment 714). The non-transitory machine readable medium 702 maycomprise a memory semiconductor (e.g., a semiconductor utilizing staticrandom access memory (SRAM), dynamic random access memory (DRAM), and/orsynchronous dynamic random access memory (SDRAM) technologies), aplatter of a hard disk drive, a flash memory device, or a magnetic oroptical disc (such as a compact disc (CD), digital versatile disc (DVD),or floppy disk). The example non-transitory machine readable medium 702stores computer-readable data 704 that, when subjected to reading 706 bya reader 710 of a device 708 (e.g., a read head of a hard disk drive, ora read operation invoked on a solid-state storage device), express theprocessor-executable instructions 712. In some embodiments, theprocessor-executable instructions 712, when executed, cause performanceof operations, such as at least some of the example method 400 of FIG. 4, for example. In some embodiments, the processor-executableinstructions 712 are configured to cause implementation of a system,such as at least some of the example system 501 of FIGS. 5A-5M and/orthe example system 601 of FIG. 6 , for example.

3. Usage of Terms

As used in this application, “component,” “module,” “system”,“interface”, and/or the like are generally intended to refer to acomputer-related entity, either hardware, a combination of hardware andsoftware, software, or software in execution. For example, a componentmay be, but is not limited to being, a process running on a processor, aprocessor, an object, an executable, a thread of execution, a program,and/or a computer. By way of illustration, both an application runningon a controller and the controller can be a component. One or morecomponents may reside within a process and/or thread of execution and acomponent may be localized on one computer and/or distributed betweentwo or more computers.

Unless specified otherwise, “first,” “second,” and/or the like are notintended to imply a temporal aspect, a spatial aspect, an ordering, etc.Rather, such terms are merely used as identifiers, names, etc. forfeatures, elements, items, etc. For example, a first object and a secondobject generally correspond to object A and object B or two different ortwo identical objects or the same object.

Moreover, “example” is used herein to mean serving as an instance,illustration, etc., and not necessarily as advantageous. As used herein,“or” is intended to mean an inclusive “or” rather than an exclusive“or”. In addition, “a” and “an” as used in this application aregenerally be construed to mean “one or more” unless specified otherwiseor clear from context to be directed to a singular form. Also, at leastone of A and B and/or the like generally means A or B or both A and B.Furthermore, to the extent that “includes”, “having”, “has”, “with”,and/or variants thereof are used in either the detailed description orthe claims, such terms are intended to be inclusive in a manner similarto the term “comprising”.

Although the subject matter has been described in language specific tostructural features and/or methodological acts, it is to be understoodthat the subject matter defined in the appended claims is notnecessarily limited to the specific features or acts described above.Rather, the specific features and acts described above are disclosed asexample forms of implementing at least some of the claims.

Furthermore, the claimed subject matter may be implemented as a method,apparatus, or article of manufacture using standard programming and/orengineering techniques to produce software, firmware, hardware, or anycombination thereof to control a computer to implement the disclosedsubject matter. The term “article of manufacture” as used herein isintended to encompass a computer program accessible from anycomputer-readable device, carrier, or media. Of course, manymodifications may be made to this configuration without departing fromthe scope or spirit of the claimed subject matter.

Various operations of embodiments are provided herein. In an embodiment,one or more of the operations described may constitute computer readableinstructions stored on one or more computer and/or machine readablemedia, which if executed will cause the operations to be performed. Theorder in which some or all of the operations are described should not beconstrued as to imply that these operations are necessarily orderdependent. Alternative ordering will be appreciated by one skilled inthe art having the benefit of this description. Further, it will beunderstood that not all operations are necessarily present in eachembodiment provided herein. Also, it will be understood that not alloperations are necessary in some embodiments.

Also, although the disclosure has been shown and described with respectto one or more implementations, equivalent alterations and modificationswill occur to others skilled in the art based upon a reading andunderstanding of this specification and the annexed drawings. Thedisclosure includes all such modifications and alterations and islimited only by the scope of the following claims. In particular regardto the various functions performed by the above described components(e.g., elements, resources, etc.), the terms used to describe suchcomponents are intended to correspond, unless otherwise indicated, toany component which performs the specified function of the describedcomponent (e.g., that is functionally equivalent), even though notstructurally equivalent to the disclosed structure. In addition, while aparticular feature of the disclosure may have been disclosed withrespect to only one of several implementations, such feature may becombined with one or more other features of the other implementations asmay be desired and advantageous for any given or particular application.

What is claimed is:
 1. A method, comprising: identifying a firstplurality of sets of event information associated with a first pluralityof events, wherein the first plurality of sets of event informationcomprises: a second plurality of sets of event information associatedwith a plurality of accidental click events of the first plurality ofevents; and a third plurality of sets of event information associatedwith a plurality of skip events of the first plurality of events;determining a plurality of accidental click probabilities associatedwith a second plurality of events comprising the plurality of accidentalclick events and the plurality of skip events, wherein the determiningthe plurality of accidental click probabilities comprises: determining afirst accidental click probability, associated with a first accidentalclick event of the plurality of accidental click events, based upon afirst set of event information associated with the first accidentalclick event; and determining a second accidental click probability,associated with a first skip event of the plurality of skip events,based upon a second set of event information associated with the firstskip event; performing machine learning model training, using the firstplurality of sets of event information associated with the firstplurality of events and a first plurality of labels associated with thefirst plurality of events, to generate a first machine learning model,wherein: the first plurality of labels comprises a second plurality oflabels associated with the second plurality of events; and labels of thesecond plurality of labels are based upon the plurality of accidentalclick probabilities and comprise: a first label, associated with thefirst accidental click event, based upon the first accidental clickprobability; and a second label, associated with the first skip event,based upon the second accidental click probability; receiving a requestfor content associated with a client device; responsive to receiving therequest for content, determining a plurality of click probabilitiesassociated with a plurality of content items using the first machinelearning model; and selecting, based upon the plurality of clickprobabilities, a first content item of the plurality of content itemsfor presentation via the client device.
 2. The method of claim 1,wherein: the determining the plurality of accidental click probabilitiesis performed using a second machine learning model.
 3. The method ofclaim 2, comprising: performing machine learning model training, using afourth plurality of sets of event information associated with a secondplurality of accidental click events and a fifth plurality of sets ofevent information associated with a second plurality of skip events, togenerate the second machine learning model.
 4. The method of claim 1,wherein the determining the plurality of click probabilities comprises:determining a first click probability associated with the first contentitem, using the first machine learning model, based upon at least one ofclient information associated with the client device, content iteminformation associated with the first content item or internet resourceinformation associated with an internet resource associated with therequest for content.
 5. The method of claim 1, comprising: presenting asecond content item, via a second client device, on a first internetresource, wherein the second content item is selected via the secondclient device during the presenting the second content item; anddetecting occurrence of the first accidental click event based upon adetermination that a dwell time associated with the selection of thesecond content item is less than a threshold dwell time, wherein thedwell time comprises a time during which a second internet resource,accessed by the second client device responsive to the selection of thesecond content item, is presented via the second client device.
 6. Themethod of claim 5, comprising: determining the dwell time based upon afirst time and a second time, wherein: the first time corresponds to atleast one of: a time at which the second content item is selected viathe second client device; or a time at which the second internetresource is accessed responsive to the selection of the second contentitem; and the second time corresponds to at least one of: a time atwhich the second client device returns to the first internet resourceafter the second content item is selected; or a time at which the secondclient device leaves the second internet resource.
 7. The method ofclaim 1, comprising: presenting a second content item, via a secondclient device, on a first internet resource; and detecting occurrence ofthe first skip event based upon a determination that the second contentitem is not selected via the second client device during the presentingthe second content item.
 8. The method of claim 7, wherein: the secondaccidental click probability is representative of a probability thatpresentation of the second content item via the second client devicewould be followed by occurrence of an accidental click event.
 9. Themethod of claim 1, comprising: presenting a second content item, via asecond client device, on a first internet resource, wherein the secondcontent item is selected via the second client device during thepresenting the second content item; and detecting occurrence of a firstintentional click event of the first plurality of events based upon adetermination that a dwell time associated with the selection of thesecond content item exceeds a threshold dwell time, wherein the dwelltime comprises a time during which a second internet resource, accessedby the second client device responsive to the selection of the secondcontent item, is presented via the second client device.
 10. The methodof claim 1, comprising: determining one or more features associated withthe request for content.
 11. The method of claim 10, wherein: thedetermining the plurality of click probabilities using the first machinelearning model is performed based upon the one or more features.
 12. Themethod of claim 1, wherein: the plurality of click probabilitiescomprises a first click probability associated with the first contentitem; and the first click probability is representative of a probabilityof receiving a selection of the first content item responsive topresenting the first content item via the client device.
 13. A computingdevice comprising: a processor; and memory comprisingprocessor-executable instructions that when executed by the processorcause performance of operations, the operations comprising: identifyinga first plurality of sets of event information associated with a firstplurality of events, wherein the first plurality of sets of eventinformation comprises: a second plurality of sets of event informationassociated with a plurality of skip events of the first plurality ofevents; determining a plurality of accidental click probabilitiesassociated with the plurality of skip events, wherein the determiningthe plurality of accidental click probabilities comprises determining afirst accidental click probability, associated with a first skip eventof the plurality of skip events, based upon a first set of eventinformation associated with the first skip event; performing machinelearning model training, using the first plurality of sets of eventinformation associated with the first plurality of events and a firstplurality of labels associated with the first plurality of events, togenerate a first machine learning model, wherein: the first plurality oflabels comprises a second plurality of labels associated with theplurality of skip events; and labels of the second plurality of labelsare based upon the plurality of accidental click probabilities andcomprise a first label, associated with the first skip event, based uponthe first accidental click probability; receiving a request for contentassociated with a client device; responsive to receiving the request forcontent, determining a plurality of click probabilities associated witha plurality of content items using the first machine learning model; andselecting, based upon the plurality of click probabilities, a firstcontent item of the plurality of content items for presentation via theclient device.
 14. The computing device of claim 13, wherein: thedetermining the plurality of accidental click probabilities is performedusing a second machine learning model.
 15. The computing device of claim14, the operations comprising: performing machine learning modeltraining, using a third plurality of sets of event informationassociated with a plurality of accidental click events and a fourthplurality of sets of event information associated with a secondplurality of skip events, to generate the second machine learning model.16. The computing device of claim 15, the operations comprising:presenting a second content item, via a second client device, on a firstinternet resource, wherein the second content item is selected via thesecond client device during the presenting the second content item; anddetecting occurrence of a first accidental click event of the pluralityof accidental click events based upon a determination that a dwell timeassociated with the selection of the second content item is less than athreshold dwell time, wherein the dwell time comprises a time duringwhich a second internet resource, accessed by the second client deviceresponsive to the selection of the second content item, is presented viathe second client device.
 17. The computing device of claim 13, whereinthe determining the plurality of click probabilities comprises:determining a first click probability associated with the first contentitem, using the first machine learning model, based upon at least one ofclient information associated with the client device, content iteminformation associated with the first content item or internet resourceinformation associated with an internet resource associated with therequest for content.
 18. The computing device of claim 13, theoperations comprising: presenting a second content item, via a secondclient device, on a first internet resource, wherein the second contentitem is selected via the second client device during the presenting thesecond content item; and detecting occurrence of a first intentionalclick event of the first plurality of events based upon a determinationthat a dwell time associated with the selection of the second contentitem exceeds a threshold dwell time, wherein the dwell time comprises atime during which a second internet resource, accessed by the secondclient device responsive to the selection of the second content item, ispresented via the second client device.
 19. A non-transitory machinereadable medium having stored thereon processor-executable instructionsthat when executed cause performance of operations, the operationscomprising: identifying a first plurality of sets of event informationassociated with a first plurality of events, wherein the first pluralityof sets of event information comprises: a second plurality of sets ofevent information associated with a plurality of accidental click eventsof the first plurality of events; determining a plurality of accidentalclick probabilities associated with the plurality of accidental clickevents, wherein the determining the plurality of accidental clickprobabilities comprises determining a first accidental clickprobability, associated with a first accidental click event of theplurality of accidental click events, based upon a first set of eventinformation associated with the first accidental click event; performingmachine learning model training, using the first plurality of sets ofevent information associated with the first plurality of events and afirst plurality of labels associated with the first plurality of events,to generate a first machine learning model, wherein: the first pluralityof labels comprises a second plurality of labels associated with theplurality of accidental click events; and labels of the second pluralityof labels are based upon the plurality of accidental click probabilitiesand comprise a first label, associated with the first accidental clickevent, based upon the first accidental click probability; receiving arequest for content associated with a client device; responsive toreceiving the request for content, determining a plurality of clickprobabilities associated with a plurality of content items using thefirst machine learning model; and selecting, based upon the plurality ofclick probabilities, a first content item of the plurality of contentitems for presentation via the client device.
 20. The non-transitorymachine readable medium of claim 19, wherein the determining theplurality of click probabilities comprises: determining a first clickprobability associated with the first content item, using the firstmachine learning model, based upon at least one of client informationassociated with the client device, content item information associatedwith the first content item or internet resource information associatedwith an internet resource associated with the request for content.